fix: missing value after comparison #63831
Conversation
The comparison with pd.NA incorrectly returned False. This fix overwrites the result with NA value in the original location. Closes pandas-dev#63328
|
Hello @rhshadrach, TestsI have tested the purposed changes with command Explain why it is written this wayI originally wanted to use result[pd.isna(x)] = pd.NA(suggest by numpy: https://numpy.org/doc/stable/user/how-to-index.html#replace-values-after-filtering), but this will raise So that's why I ultimately chose the current method. I think a test is also required to check the new behaviour, but I wanted to first make sure that this is the right approach and won't cause more problems. |
rhshadrach
left a comment
rhshadrach
left a comment
There was a problem hiding this comment.
Thanks for the PR. When changing the behavior, please add tests and a note in the whatsnew of the next minor release (3.1 here).
| result = libops.vec_compare(x.ravel(), y.ravel(), op) | ||
| else: | ||
| result = libops.scalar_compare(x.ravel(), y, op) | ||
| result = np.where(isna(x), x, result) |
There was a problem hiding this comment.
This is not the correct behavior; e.g. 5 = = np.nan should give False. We only want to change the behavior for pd.NA.
There was a problem hiding this comment.
I wanted to make sure, only replace pd.NA is the intended behavior? so not to replace any of None, pd.NaT, np.nan?
If so, pd.isna() and np.isnan() both aren't gonna work. It seems only the Python is operator is capable to differentiate pd.NA with others. If there's another way to do this, please let me know. Much appreciated!
2 participants
The comparison with pd.NA incorrectly returned False. This fix overwrites the result with NA value in the original location.
Closes #63328
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.AGENTS.md.